{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ALBERT\n",
    "[ALBERT](https://arxiv.org/pdf/1909.11942.pdf),A Lite Bert    \n",
    "参考链接：   \n",
    "https://amitness.com/2020/02/albert-visual-summary/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BERT 的缺点\n",
    "内存限制和通讯开销：\n",
    "- BERT非常大的模型，巨量的训练参数，想要从头开始训练，需要大量的计算资源，且计算机内存会限制模型大小\n",
    "- 一种解决方法就是分布式训练；数据被拆分到多个机器上，单独进行训练，然后再同步不同的梯度，大量参数的同步过程造成较大的通讯开销，延缓训练过程；同理 模型并行化 时，也会造成同样的瓶颈 \n",
    "\n",
    "模型退化：\n",
    "- 模型变大，并不一定会获得更好的性能，反而性能可能下降"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ALBERT的改进\n",
    " ALBERT 本质就是对 BERT 模型压缩后的产物。\n",
    " > 模型压缩有很多手段，包括剪枝，参数共享，低秩分解，网络结构设计，知识蒸馏等\n",
    " \n",
    " ALBERT降低了模型参数量从而降低了模型的训练时间（通信开销降低），但是，**模型的预测推理时间并没有降低**。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cross-layer parameter sharing\n",
    "- BERT-LARGE 有 24 层，而 BERT-BASE 有 12 层；随着层数增加，参数量指数级增加   \n",
    "      \n",
    "      \n",
    "- ALBERT 提出了 跨层参数共享 的概念，不是单独学习 BERT 每一层的参数，而是将每一层的参数设置为相同\n",
    "    - 可以只共享 feed-forward 层的参数，或 attention 层的参数，或整个编码层的参数\n",
    "    - 共享整层参数时，模型参数量明显下降，从 110M 下降到 31M \n",
    "    \n",
    "- 参数共享对性能的影响很小，且能够帮助模型稳定网络的参数。这点是通过L2距离与 cosine similarity 得出的"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SOP 替代 NSP\n",
    "- SOP 全称为 Sentence Order Prediction，其用来取代 NSP 在 BERT 中的作用，\n",
    "    - ROBERTA 和 XLNET 模型显示 NSP 非但没有作用，反而会对模型带来一些损害。删除 NSP 任务，模型在一些列任务上的性能会提升\n",
    "    - NSP 相比 MLM 并不是一个难的任务，NSP 设置到主题预测和连贯性预测，主题预测比较容易学习，因为与 MLM 任务重合；因此就算没有学到 连贯性预测，NSP 也会得到较高的精度\n",
    "    \n",
    "      \n",
    "- SOP的方式与NSP相似，也是判断第二句话是不是第一句话的下一句，但对于负例来说，SOP并不从不相关的句子中生成，而是将原来连续的两句话翻转形成负例。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Factorized embedding parameterization\n",
    "\n",
    "通过对Embedding 部分降维来达到降低参数的作用。\n",
    "- BERT-Base 中 Embedding 层的维度与隐层的维度一样都是768；  \n",
    "- 而对于词的分布式表示，如 Word2Vec 就多采用50或300这样的维度。\n",
    "- 通过将 Embedding 部分分解来达到降低参数量的作用，其以公式表示如下：\n",
    "\n",
    "$$O(V\\times H) \\rightarrow O(V\\times E+E\\times H)$$\n",
    "\n",
    "    V：词表大小；H：隐层维度；E：词向量维度\n",
    "\n",
    "以 BERT-Base 为例，    \n",
    "- 词表大小为3w，此时的参数量为：768 * 3w = 23040000。 \n",
    "- 如果将 Embedding 的维度改为 128，那么此时Embedding层的参数量为： 128 * 3w + 128 * 768 = 3938304。\n",
    "- Embedding参数量从原来的23M变为了现在的4M，\n",
    "- 但从整个模型来看，BERT-Base的参数量在 110M，降低19M也不能产生什么革命性的变化。\n",
    "- 可以说 Embedding 层的因式分解其实并不是降低参数量的主要手段。\n",
    "- 意忽略了Position Embedding的那部分参数量， 主要是考虑到512相对于3W显得有点微不足道。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-09T10:11:47.753579Z",
     "start_time": "2020-05-09T10:11:47.748145Z"
    }
   },
   "source": [
    "### ALBERT 的性能提升\n",
    "- 参数量比 BERT-large 小 18 倍\n",
    "- 训练比 BERT 快 1.7 倍\n",
    "- 在 GLUE, RACE 和 SQUAD 任务上获得最佳性能"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ALBERT 中文实体识别"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 分词器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T03:37:21.522598Z",
     "start_time": "2020-05-29T03:37:20.279760Z"
    }
   },
   "outputs": [],
   "source": [
    "from transformers import BertTokenizer, AlbertForTokenClassification\n",
    "model_path = \"../../H/models/huggingface/torch/albert_chinese_xlarge/\"\n",
    "tokenizer = BertTokenizer.from_pretrained(model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据集\n",
    "预料来源：https://github.com/InsaneLife/ChineseNLPCorpus/tree/master/NER/renMinRiBao"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T03:37:22.280750Z",
     "start_time": "2020-05-29T03:37:21.523766Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "文本序列的数量： 37924\n",
      "所有标签： {'B_nt', 'M_nr', 'B_nr', 'E_ns', 'E_nr', 'B_ns', 'M_ns', 'O', 'E_nt', 'M_nt'}\n",
      "{0: 'B_nt', 1: 'M_nr', 2: 'B_nr', 3: 'E_ns', 4: 'E_nr', 5: 'B_ns', 6: 'M_ns', 7: 'O', 8: 'E_nt', 9: 'M_nt'}\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "import codecs\n",
    "\n",
    "input_data = codecs.open('../datasets/ner/renmin/renmin4.txt', 'r', 'utf-8')\n",
    "\n",
    "# 1. 将标注子句 拆分成 字列表 和 对应的标注列表 #############\n",
    "#####################################################\n",
    "datas = []\n",
    "labels = []\n",
    "\n",
    "# 表征该处的标签是填充的,应不应该使用？？？？？？\n",
    "# tags = set(['PAD']) \n",
    "tags = set()\n",
    "\n",
    "for line in input_data.readlines():\n",
    "    linedata = list()\n",
    "    linelabel = list()\n",
    "\n",
    "    line = line.split()\n",
    "\n",
    "    numNotO = 0\n",
    "    for word in line:\n",
    "        word = word.split('/')\n",
    "        linedata.append(word[0])\n",
    "        linelabel.append(word[1])\n",
    "\n",
    "        tags.add(word[1])\n",
    "\n",
    "        if word[1] != 'O':  # 标注全为 O 的子句\n",
    "            numNotO += 1\n",
    "\n",
    "    if numNotO != 0:  # 只保存 标注不全为 O 的子句\n",
    "        datas.append(linedata)\n",
    "        labels.append(linelabel)\n",
    "\n",
    "input_data.close()\n",
    "print(\"文本序列的数量：\", len(datas))  # 字列表 组成的列表\n",
    "assert (len(labels) == len(datas))  # 对应的 标注列表 组成的列表\n",
    "\n",
    "# 2. 创建标签字典 ################################\n",
    "#####################################################\n",
    "\n",
    "print(\"所有标签：\", tags)\n",
    "tag2id = {tag: i for i, tag in enumerate(tags)}\n",
    "\n",
    "id2tag = {i: tag for tag, i in tag2id.items()}\n",
    "print(id2tag)\n",
    "\n",
    "print(\"-\" * 80)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T03:37:22.286997Z",
     "start_time": "2020-05-29T03:37:22.281841Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'B_nt': 0,\n",
       " 'M_nr': 1,\n",
       " 'B_nr': 2,\n",
       " 'E_ns': 3,\n",
       " 'E_nr': 4,\n",
       " 'B_ns': 5,\n",
       " 'M_ns': 6,\n",
       " 'O': 7,\n",
       " 'E_nt': 8,\n",
       " 'M_nt': 9}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tag2id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T03:37:22.291772Z",
     "start_time": "2020-05-29T03:37:22.288103Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "({'B_ns': 0,\n",
       "  'M_ns': 1,\n",
       "  'E_ns': 2,\n",
       "  'B_nr': 3,\n",
       "  'M_nr': 4,\n",
       "  'E_nr': 5,\n",
       "  'B_nt': 6,\n",
       "  'M_nt': 7,\n",
       "  'E_nt': 8,\n",
       "  'O': 9},\n",
       " {0: 'B_ns',\n",
       "  1: 'M_ns',\n",
       "  2: 'E_ns',\n",
       "  3: 'B_nr',\n",
       "  4: 'M_nr',\n",
       "  5: 'E_nr',\n",
       "  6: 'B_nt',\n",
       "  7: 'M_nt',\n",
       "  8: 'E_nt',\n",
       "  9: 'O'})"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 为了便于理解，将标签排序\n",
    "\n",
    "tags = [\n",
    "    'B_ns', 'M_ns', 'E_ns', 'B_nr', 'M_nr', 'E_nr', 'B_nt', 'M_nt', 'E_nt',\n",
    "    'O',\n",
    "]\n",
    "tag2id = {tag: idx for idx, tag in enumerate(tags)}\n",
    "id2tag = {idx: tag for idx, tag in enumerate(tags)}\n",
    "tag2id, id2tag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T03:37:22.359934Z",
     "start_time": "2020-05-29T03:37:22.292641Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "各类标签出现数量：\n",
      "B_ns : 22427\n",
      "M_ns : 13579\n",
      "E_ns : 22427\n",
      "B_nr : 19981\n",
      "M_nr : 18235\n",
      "E_nr : 19981\n",
      "B_nt : 10834\n",
      "M_nt : 40955\n",
      "E_nt : 10834\n",
      "O : 389034\n",
      "总标签数： 568287\n"
     ]
    }
   ],
   "source": [
    "tags_count = {}\n",
    "for seq in labels:\n",
    "    for tag in seq:\n",
    "        tags_count[tag] = tags_count.get(tag, 0) + 1\n",
    "print(\"各类标签出现数量：\")\n",
    "for tag in tags:\n",
    "    if tag in tags_count:\n",
    "        print(tag, \":\", tags_count[tag])\n",
    "print(\"总标签数：\", sum([len(seq) for seq in labels]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T03:37:22.365089Z",
     "start_time": "2020-05-29T03:37:22.360708Z"
    }
   },
   "outputs": [],
   "source": [
    "# 处理成等长\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "def pad_sequences(sequences,\n",
    "                  maxlen=None,\n",
    "                  dtype='int64',\n",
    "                  padding='post',\n",
    "                  truncating='post',\n",
    "                  value=0.):\n",
    "\n",
    "    num_samples = len(sequences)\n",
    "    lengths = [len(sample) for sample in sequences]\n",
    "\n",
    "    if maxlen is None:\n",
    "        maxlen = np.max(lengths)\n",
    "\n",
    "    x = np.full((num_samples, maxlen), value, dtype=dtype)\n",
    "    for idx, s in enumerate(sequences):\n",
    "        if not len(s):\n",
    "            continue  # empty list/array was found\n",
    "        if truncating == 'pre':\n",
    "            trunc = s[-maxlen:]\n",
    "        elif truncating == 'post':\n",
    "            trunc = s[:maxlen]\n",
    "        else:\n",
    "            raise ValueError('Truncating type \"%s\" '\n",
    "                             'not understood' % truncating)\n",
    "\n",
    "        trunc = np.asarray(trunc, dtype=dtype)\n",
    "\n",
    "        if padding == 'post':\n",
    "            x[idx, :len(trunc)] = trunc\n",
    "        elif padding == 'pre':\n",
    "            x[idx, -len(trunc):] = trunc\n",
    "        else:\n",
    "            raise ValueError('Padding type \"%s\" not understood' % padding)\n",
    "    return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T03:37:22.701964Z",
     "start_time": "2020-05-29T03:37:22.365833Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(37924, 60)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 输入向量\n",
    "input_ids = pad_sequences(\n",
    "    [tokenizer.convert_tokens_to_ids(seq) for seq in datas],\n",
    "    maxlen = 60,\n",
    ")\n",
    "input_ids.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T03:37:22.705416Z",
     "start_time": "2020-05-29T03:37:22.702872Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 704, 1066,  704, 1925, 2600,  741, 6381,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_ids[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T03:37:22.709140Z",
     "start_time": "2020-05-29T03:37:22.707084Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['在', '一', '个', '中', '国', '的', '原', '则', '下']\n",
      "['在', '一', '个', '中', '国', '的', '原', '则', '下', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]']\n"
     ]
    }
   ],
   "source": [
    "# 检查是否正确\n",
    "print(datas[999])\n",
    "print(tokenizer.convert_ids_to_tokens(input_ids[999]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T03:37:22.906376Z",
     "start_time": "2020-05-29T03:37:22.710183Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(37924, 60)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 训练目标序列\n",
    "tags = pad_sequences(\n",
    "    [[tag2id[l] for l in seq] for seq in labels],\n",
    "    maxlen=60,\n",
    "    value=0.,\n",
    ")\n",
    "tags.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T03:37:22.909757Z",
     "start_time": "2020-05-29T03:37:22.907283Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([6, 7, 7, 8, 9, 9, 9, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tags[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T03:37:22.913001Z",
     "start_time": "2020-05-29T03:37:22.910665Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['O', 'O', 'O', 'B_ns', 'E_ns', 'O', 'O', 'O', 'O']\n",
      "['O', 'O', 'O', 'B_ns', 'E_ns', 'O', 'O', 'O', 'O', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns', 'B_ns']\n"
     ]
    }
   ],
   "source": [
    "print(labels[999])\n",
    "print([id2tag[idx] for idx in tags[999]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T03:37:22.920870Z",
     "start_time": "2020-05-29T03:37:22.913840Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0.])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 掩码，表征哪些元素是填充的\n",
    "\n",
    "masks = (input_ids != 0).astype(np.float)  # float 类型\n",
    "masks\n",
    "masks[999]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T03:37:22.939237Z",
     "start_time": "2020-05-29T03:37:22.921761Z"
    }
   },
   "outputs": [],
   "source": [
    "# 拆分数据集，训练、验证、测试\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "tr_inputs, val_inputs, tr_tags, val_tags, tr_masks, val_masks = train_test_split(\n",
    "    input_ids,\n",
    "    tags,\n",
    "    masks,\n",
    "    random_state=2018,\n",
    "    test_size=0.25,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T03:37:22.942365Z",
     "start_time": "2020-05-29T03:37:22.940086Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((28443, 60), (9481, 60))"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tr_inputs.shape, val_inputs.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T03:37:22.968794Z",
     "start_time": "2020-05-29T03:37:22.943061Z"
    }
   },
   "outputs": [],
   "source": [
    "# 转换成 torch 张量\n",
    "\n",
    "import torch\n",
    "\n",
    "# input_ids ，数据类型：torch.LongTensor，形状：(batch_size, sequence_length)\n",
    "tr_inputs = torch.tensor(tr_inputs)\n",
    "val_inputs = torch.tensor(val_inputs)\n",
    "\n",
    "# labels，数据类型：torch.LongTensor，形状：(batch_size, sequence_length)\n",
    "tr_tags = torch.tensor(tr_tags)\n",
    "val_tags = torch.tensor(val_tags)\n",
    "\n",
    "# attention_mask，数据类型：torch.FloatTensor，形状：(batch_size, sequence_length)\n",
    "tr_masks = torch.tensor(tr_masks)\n",
    "val_masks = torch.tensor(val_masks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T03:37:22.971630Z",
     "start_time": "2020-05-29T03:37:22.969602Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([28443, 60]), torch.Size([9481, 60]))"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tr_inputs.shape, val_inputs.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T03:37:22.974833Z",
     "start_time": "2020-05-29T03:37:22.972364Z"
    }
   },
   "outputs": [],
   "source": [
    "# 创建批量数据集\n",
    "from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler\n",
    "\n",
    "bs = 32\n",
    "train_data = TensorDataset(tr_inputs, tr_masks, tr_tags)\n",
    "train_sampler = RandomSampler(train_data)\n",
    "train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=bs)\n",
    "\n",
    "valid_data = TensorDataset(val_inputs, val_masks, val_tags)\n",
    "valid_sampler = SequentialSampler(valid_data)\n",
    "valid_dataloader = DataLoader(valid_data, sampler=valid_sampler, batch_size=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T03:37:22.977330Z",
     "start_time": "2020-05-29T03:37:22.975508Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "889 28448\n"
     ]
    }
   ],
   "source": [
    "print(len(train_dataloader), bs * len(train_dataloader))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T03:37:22.979871Z",
     "start_time": "2020-05-29T03:37:22.978067Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "297 9504\n"
     ]
    }
   ],
   "source": [
    "print(len(valid_dataloader), bs * len(valid_dataloader))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 创建模型\n",
    "模型引用：https://huggingface.co/voidful/albert_chinese_xlarge#"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T03:37:22.981972Z",
     "start_time": "2020-05-29T03:37:22.980560Z"
    }
   },
   "outputs": [],
   "source": [
    "from transformers import AlbertForTokenClassification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T03:37:24.168658Z",
     "start_time": "2020-05-29T03:37:22.982688Z"
    }
   },
   "outputs": [],
   "source": [
    "tokenizer = BertTokenizer.from_pretrained(model_path)\n",
    "model = AlbertForTokenClassification.from_pretrained(\n",
    "    model_path,\n",
    "    num_labels=len(id2tag),\n",
    "    output_attentions=False,\n",
    "    output_hidden_states=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T03:37:25.107807Z",
     "start_time": "2020-05-29T03:37:24.169482Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AlbertForTokenClassification(\n",
       "  (albert): AlbertModel(\n",
       "    (embeddings): AlbertEmbeddings(\n",
       "      (word_embeddings): Embedding(21128, 128, padding_idx=0)\n",
       "      (position_embeddings): Embedding(512, 128)\n",
       "      (token_type_embeddings): Embedding(2, 128)\n",
       "      (LayerNorm): LayerNorm((128,), eps=1e-12, elementwise_affine=True)\n",
       "      (dropout): Dropout(p=0, inplace=False)\n",
       "    )\n",
       "    (encoder): AlbertTransformer(\n",
       "      (embedding_hidden_mapping_in): Linear(in_features=128, out_features=2048, bias=True)\n",
       "      (albert_layer_groups): ModuleList(\n",
       "        (0): AlbertLayerGroup(\n",
       "          (albert_layers): ModuleList(\n",
       "            (0): AlbertLayer(\n",
       "              (full_layer_layer_norm): LayerNorm((2048,), eps=1e-12, elementwise_affine=True)\n",
       "              (attention): AlbertAttention(\n",
       "                (query): Linear(in_features=2048, out_features=2048, bias=True)\n",
       "                (key): Linear(in_features=2048, out_features=2048, bias=True)\n",
       "                (value): Linear(in_features=2048, out_features=2048, bias=True)\n",
       "                (dropout): Dropout(p=0, inplace=False)\n",
       "                (dense): Linear(in_features=2048, out_features=2048, bias=True)\n",
       "                (LayerNorm): LayerNorm((2048,), eps=1e-12, elementwise_affine=True)\n",
       "              )\n",
       "              (ffn): Linear(in_features=2048, out_features=8192, bias=True)\n",
       "              (ffn_output): Linear(in_features=8192, out_features=2048, bias=True)\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "    )\n",
       "    (pooler): Linear(in_features=2048, out_features=2048, bias=True)\n",
       "    (pooler_activation): Tanh()\n",
       "  )\n",
       "  (dropout): Dropout(p=0, inplace=False)\n",
       "  (classifier): Linear(in_features=2048, out_features=10, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.cuda()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-28T07:23:28.283577Z",
     "start_time": "2020-05-28T07:23:27.347477Z"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-28T03:57:35.665527Z",
     "start_time": "2020-05-28T03:57:35.660579Z"
    }
   },
   "source": [
    "## 训练模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ZhAlbert(nn.Module):\n",
    "    def __init__(self, model_path, ff_dim, num_tags, dropout):\n",
    "        self.num_tags = num_tags\n",
    "        \n",
    "        self.albert = AlbertModel.from_pretrained(model_path)\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        self.linear = nn.Linear(ff_dim, num_tags)\n",
    "    \n",
    "    def forward(self, x, mask, y):\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T03:37:33.695469Z",
     "start_time": "2020-05-29T03:37:33.691698Z"
    }
   },
   "outputs": [],
   "source": [
    "import transformers\n",
    "from transformers import BertForTokenClassification, AdamW\n",
    "\n",
    "# 1. 优调整个模型，bert 模型及其上的分类层\n",
    "\n",
    "FULL_FINETUNING = True\n",
    "if FULL_FINETUNING:\n",
    "    param_optimizer = list(model.named_parameters())\n",
    "    no_decay = ['bias', 'gamma', 'beta']\n",
    "    optimizer_grouped_parameters = [{\n",
    "        'params':\n",
    "        [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],\n",
    "        'weight_decay_rate':\n",
    "        0.01\n",
    "    }, {\n",
    "        'params':\n",
    "        [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],\n",
    "        'weight_decay_rate':\n",
    "        0.0\n",
    "    }]\n",
    "\n",
    "# 2. 仅仅训练最顶层的分类层    \n",
    "else:\n",
    "    param_optimizer = list(model.classifier.named_parameters())\n",
    "    optimizer_grouped_parameters = [{\n",
    "        \"params\": [p for n, p in param_optimizer]\n",
    "    }]\n",
    "\n",
    "# 优化器    \n",
    "optimizer = AdamW(optimizer_grouped_parameters, lr=3e-5, eps=1e-8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T04:58:00.121797Z",
     "start_time": "2020-05-29T04:58:00.116600Z"
    }
   },
   "outputs": [],
   "source": [
    "from transformers import get_linear_schedule_with_warmup\n",
    "\n",
    "epochs = 20\n",
    "max_grad_norm = 1.0\n",
    "\n",
    "# 总的训练次数\n",
    "total_steps = len(train_dataloader) * epochs\n",
    "\n",
    "# 学习率规划\n",
    "scheduler = get_linear_schedule_with_warmup(\n",
    "    optimizer,\n",
    "    num_warmup_steps=0,\n",
    "    num_training_steps=total_steps,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T04:58:01.384574Z",
     "start_time": "2020-05-29T04:58:01.382477Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import f1_score, accuracy_score, classification_report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T11:24:03.662543Z",
     "start_time": "2020-05-29T04:58:02.423101Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Epoch:   0%|          | 0/20 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0, Step: 0, Train loss: 0.5352237224578857\n",
      "Epoch: 0, Step: 100, Train loss: 0.42720396300353625\n",
      "Epoch: 0, Step: 200, Train loss: 0.43819765605736727\n",
      "Epoch: 0, Step: 300, Train loss: 0.43702682377492075\n",
      "Epoch: 0, Step: 400, Train loss: 0.4368162702889811\n",
      "Epoch: 0, Step: 500, Train loss: 0.43662129369324554\n",
      "Epoch: 0, Step: 600, Train loss: 0.43581701044036625\n",
      "Epoch: 0, Step: 700, Train loss: 0.433231198753678\n",
      "Epoch: 0, Step: 800, Train loss: 0.4292947600843308\n",
      "Epoch: 0, Average train loss: 0.42743512190233063 \n",
      "Validation loss: 0.39762468525656947 at epoch 0\n",
      "Validation Accuracy: 0.8272046281555606at epoch 0\n",
      "Validation F1-Score: 0.49681934244747694at epoch 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Epoch:   5%|▌         | 1/20 [19:16<6:06:17, 1156.73s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "        B_nr       0.35      0.58      0.43      4839\n",
      "        B_ns       0.33      0.25      0.28      5576\n",
      "        B_nt       0.55      0.17      0.26      2695\n",
      "        E_nr       0.35      0.32      0.33      4838\n",
      "        E_ns       0.45      0.53      0.49      5574\n",
      "        E_nt       0.64      0.81      0.72      2692\n",
      "        M_nr       0.58      0.17      0.27      4382\n",
      "        M_ns       0.44      0.47      0.45      3453\n",
      "        M_nt       0.72      0.80      0.76     10073\n",
      "           O       0.98      0.98      0.98     96582\n",
      "\n",
      "    accuracy                           0.83    140704\n",
      "   macro avg       0.54      0.51      0.50    140704\n",
      "weighted avg       0.83      0.83      0.82    140704\n",
      "\n",
      "\n",
      "Epoch: 1, Step: 0, Train loss: 0.32099899649620056\n",
      "Epoch: 1, Step: 100, Train loss: 0.38655441940420926\n",
      "Epoch: 1, Step: 200, Train loss: 0.3707519302617258\n",
      "Epoch: 1, Step: 300, Train loss: 0.36644711590486506\n",
      "Epoch: 1, Step: 400, Train loss: 0.36531822186455765\n",
      "Epoch: 1, Step: 500, Train loss: 0.36380910031690805\n",
      "Epoch: 1, Step: 600, Train loss: 0.36096755208171744\n",
      "Epoch: 1, Step: 700, Train loss: 0.3570009306944726\n",
      "Epoch: 1, Step: 800, Train loss: 0.34782256375761067\n",
      "Epoch: 1, Average train loss: 0.33415379319905697 \n",
      "Validation loss: 0.23264556972667425 at epoch 1\n",
      "Validation Accuracy: 0.9336479417784853at epoch 1\n",
      "Validation F1-Score: 0.8389981004375622at epoch 1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Epoch:  10%|█         | 2/20 [38:36<5:47:15, 1157.51s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "        B_nr       0.79      0.85      0.82      4839\n",
      "        B_ns       0.91      0.84      0.88      5576\n",
      "        B_nt       0.85      0.88      0.86      2695\n",
      "        E_nr       0.86      0.69      0.76      4838\n",
      "        E_ns       0.84      0.79      0.82      5574\n",
      "        E_nt       0.84      0.84      0.84      2692\n",
      "        M_nr       0.82      0.79      0.81      4382\n",
      "        M_ns       0.70      0.79      0.74      3453\n",
      "        M_nt       0.87      0.89      0.88     10073\n",
      "           O       0.98      0.98      0.98     96582\n",
      "\n",
      "    accuracy                           0.93    140704\n",
      "   macro avg       0.85      0.84      0.84    140704\n",
      "weighted avg       0.93      0.93      0.93    140704\n",
      "\n",
      "\n",
      "Epoch: 2, Step: 0, Train loss: 0.16232149302959442\n",
      "Epoch: 2, Step: 100, Train loss: 0.17172129481735796\n",
      "Epoch: 2, Step: 200, Train loss: 0.1650525854335199\n",
      "Epoch: 2, Step: 300, Train loss: 0.1634592230546217\n",
      "Epoch: 2, Step: 400, Train loss: 0.16153084214061425\n",
      "Epoch: 2, Step: 500, Train loss: 0.1586315475902455\n",
      "Epoch: 2, Step: 600, Train loss: 0.1550974911683163\n",
      "Epoch: 2, Step: 700, Train loss: 0.1535539394443836\n",
      "Epoch: 2, Step: 800, Train loss: 0.15093585436049844\n",
      "Epoch: 2, Average train loss: 0.14960905539313973 \n",
      "Validation loss: 0.16300472438410676 at epoch 2\n",
      "Validation Accuracy: 0.9588213554696384at epoch 2\n",
      "Validation F1-Score: 0.9105317468265539at epoch 2\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Epoch:  15%|█▌        | 3/20 [57:55<5:28:05, 1157.98s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "        B_nr       0.95      0.96      0.95      4839\n",
      "        B_ns       0.93      0.92      0.92      5576\n",
      "        B_nt       0.90      0.90      0.90      2695\n",
      "        E_nr       0.90      0.95      0.93      4838\n",
      "        E_ns       0.90      0.90      0.90      5574\n",
      "        E_nt       0.85      0.91      0.88      2692\n",
      "        M_nr       0.94      0.95      0.94      4382\n",
      "        M_ns       0.89      0.74      0.81      3453\n",
      "        M_nt       0.88      0.91      0.89     10073\n",
      "           O       0.98      0.98      0.98     96582\n",
      "\n",
      "    accuracy                           0.96    140704\n",
      "   macro avg       0.91      0.91      0.91    140704\n",
      "weighted avg       0.96      0.96      0.96    140704\n",
      "\n",
      "\n",
      "Epoch: 3, Step: 0, Train loss: 0.08002409338951111\n",
      "Epoch: 3, Step: 100, Train loss: 0.11343243734745106\n",
      "Epoch: 3, Step: 200, Train loss: 0.11030179945594487\n",
      "Epoch: 3, Step: 300, Train loss: 0.10515119568243672\n",
      "Epoch: 3, Step: 400, Train loss: 0.1075554807505517\n",
      "Epoch: 3, Step: 500, Train loss: 0.1070954875682226\n",
      "Epoch: 3, Step: 600, Train loss: 0.10565225892822674\n",
      "Epoch: 3, Step: 700, Train loss: 0.10372685189206106\n",
      "Epoch: 3, Step: 800, Train loss: 0.10277210510034723\n",
      "Epoch: 3, Average train loss: 0.10223721776626946 \n",
      "Validation loss: 0.14325643132161664 at epoch 3\n",
      "Validation Accuracy: 0.9663762224243803at epoch 3\n",
      "Validation F1-Score: 0.9285535470793855at epoch 3\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Epoch:  20%|██        | 4/20 [1:17:13<5:08:50, 1158.18s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "        B_nr       0.95      0.97      0.96      4839\n",
      "        B_ns       0.94      0.91      0.93      5576\n",
      "        B_nt       0.88      0.93      0.91      2695\n",
      "        E_nr       0.94      0.96      0.95      4838\n",
      "        E_ns       0.95      0.87      0.91      5574\n",
      "        E_nt       0.89      0.94      0.92      2692\n",
      "        M_nr       0.96      0.96      0.96      4382\n",
      "        M_ns       0.85      0.85      0.85      3453\n",
      "        M_nt       0.90      0.94      0.92     10073\n",
      "           O       0.99      0.98      0.99     96582\n",
      "\n",
      "    accuracy                           0.97    140704\n",
      "   macro avg       0.93      0.93      0.93    140704\n",
      "weighted avg       0.97      0.97      0.97    140704\n",
      "\n",
      "\n",
      "Epoch: 4, Step: 0, Train loss: 0.004194826819002628\n",
      "Epoch: 4, Step: 100, Train loss: 0.08078027643853485\n",
      "Epoch: 4, Step: 200, Train loss: 0.08059446531259895\n",
      "Epoch: 4, Step: 300, Train loss: 0.07731485844549546\n",
      "Epoch: 4, Step: 400, Train loss: 0.07669327109227621\n",
      "Epoch: 4, Step: 500, Train loss: 0.07656195740115099\n",
      "Epoch: 4, Step: 600, Train loss: 0.07643920291848977\n",
      "Epoch: 4, Step: 700, Train loss: 0.07569832256769286\n",
      "Epoch: 4, Step: 800, Train loss: 0.07598005211330745\n",
      "Epoch: 4, Average train loss: 0.07635904061272782 \n",
      "Validation loss: 0.13162750811300394 at epoch 4\n",
      "Validation Accuracy: 0.968572322037753at epoch 4\n",
      "Validation F1-Score: 0.9330172500906823at epoch 4\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Epoch:  25%|██▌       | 5/20 [1:36:32<4:49:34, 1158.31s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "        B_nr       0.95      0.98      0.96      4839\n",
      "        B_ns       0.96      0.92      0.94      5576\n",
      "        B_nt       0.92      0.94      0.93      2695\n",
      "        E_nr       0.95      0.96      0.95      4838\n",
      "        E_ns       0.93      0.90      0.92      5574\n",
      "        E_nt       0.90      0.91      0.91      2692\n",
      "        M_nr       0.96      0.95      0.96      4382\n",
      "        M_ns       0.86      0.87      0.86      3453\n",
      "        M_nt       0.90      0.94      0.92     10073\n",
      "           O       0.99      0.98      0.99     96582\n",
      "\n",
      "    accuracy                           0.97    140704\n",
      "   macro avg       0.93      0.94      0.93    140704\n",
      "weighted avg       0.97      0.97      0.97    140704\n",
      "\n",
      "\n",
      "Epoch: 5, Step: 0, Train loss: 0.03187444806098938\n",
      "Epoch: 5, Step: 100, Train loss: 0.05894992280622373\n",
      "Epoch: 5, Step: 200, Train loss: 0.05865482999062842\n",
      "Epoch: 5, Step: 300, Train loss: 0.05919558091624408\n",
      "Epoch: 5, Step: 400, Train loss: 0.0580424452432782\n",
      "Epoch: 5, Step: 500, Train loss: 0.057810549911003775\n",
      "Epoch: 5, Step: 600, Train loss: 0.05797222692530274\n",
      "Epoch: 5, Step: 700, Train loss: 0.05974872462974912\n",
      "Epoch: 5, Step: 800, Train loss: 0.05921147151712965\n",
      "Epoch: 5, Average train loss: 0.0583373357707381 \n",
      "Validation loss: 0.14087198332253129 at epoch 5\n",
      "Validation Accuracy: 0.9700790311576074at epoch 5\n",
      "Validation F1-Score: 0.9383290957491489at epoch 5\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Epoch:  30%|███       | 6/20 [1:55:51<4:30:19, 1158.54s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "        B_nr       0.97      0.98      0.97      4839\n",
      "        B_ns       0.94      0.95      0.94      5576\n",
      "        B_nt       0.92      0.93      0.93      2695\n",
      "        E_nr       0.94      0.96      0.95      4838\n",
      "        E_ns       0.90      0.94      0.92      5574\n",
      "        E_nt       0.91      0.93      0.92      2692\n",
      "        M_nr       0.97      0.97      0.97      4382\n",
      "        M_ns       0.83      0.90      0.86      3453\n",
      "        M_nt       0.92      0.93      0.93     10073\n",
      "           O       0.99      0.98      0.99     96582\n",
      "\n",
      "    accuracy                           0.97    140704\n",
      "   macro avg       0.93      0.95      0.94    140704\n",
      "weighted avg       0.97      0.97      0.97    140704\n",
      "\n",
      "\n",
      "Epoch: 6, Step: 0, Train loss: 0.017148686572909355\n",
      "Epoch: 6, Step: 100, Train loss: 0.04474899180274973\n",
      "Epoch: 6, Step: 200, Train loss: 0.04890611589902465\n",
      "Epoch: 6, Step: 300, Train loss: 0.04962078881631002\n",
      "Epoch: 6, Step: 400, Train loss: 0.049914463081494664\n",
      "Epoch: 6, Step: 500, Train loss: 0.051521916659744066\n",
      "Epoch: 6, Step: 600, Train loss: 0.05055923465451074\n",
      "Epoch: 6, Step: 700, Train loss: 0.05072453593089247\n",
      "Epoch: 6, Step: 800, Train loss: 0.0489415384045094\n",
      "Epoch: 6, Average train loss: 0.04908466849125674 \n",
      "Validation loss: 0.1292283520688889 at epoch 6\n",
      "Validation Accuracy: 0.9716994541733at epoch 6\n",
      "Validation F1-Score: 0.9425766361432156at epoch 6\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Epoch:  35%|███▌      | 7/20 [2:15:09<4:10:59, 1158.41s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "        B_nr       0.98      0.96      0.97      4839\n",
      "        B_ns       0.94      0.95      0.94      5576\n",
      "        B_nt       0.94      0.93      0.93      2695\n",
      "        E_nr       0.97      0.95      0.96      4838\n",
      "        E_ns       0.93      0.92      0.93      5574\n",
      "        E_nt       0.93      0.92      0.93      2692\n",
      "        M_nr       0.98      0.96      0.97      4382\n",
      "        M_ns       0.90      0.86      0.88      3453\n",
      "        M_nt       0.93      0.93      0.93     10073\n",
      "           O       0.98      0.99      0.99     96582\n",
      "\n",
      "    accuracy                           0.97    140704\n",
      "   macro avg       0.95      0.94      0.94    140704\n",
      "weighted avg       0.97      0.97      0.97    140704\n",
      "\n",
      "\n",
      "Epoch: 7, Step: 0, Train loss: 0.03858230635523796\n",
      "Epoch: 7, Step: 100, Train loss: 0.03429088222512994\n",
      "Epoch: 7, Step: 200, Train loss: 0.04127531766541654\n",
      "Epoch: 7, Step: 300, Train loss: 0.039859507077039405\n",
      "Epoch: 7, Step: 400, Train loss: 0.0397762069083806\n",
      "Epoch: 7, Step: 500, Train loss: 0.03887080853203583\n",
      "Epoch: 7, Step: 600, Train loss: 0.040280035920650346\n",
      "Epoch: 7, Step: 700, Train loss: 0.03935857475738568\n",
      "Epoch: 7, Step: 800, Train loss: 0.04080109424571089\n",
      "Epoch: 7, Average train loss: 0.0405296498937878 \n",
      "Validation loss: 0.12203543860016873 at epoch 7\n",
      "Validation Accuracy: 0.9736396975210371at epoch 7\n",
      "Validation F1-Score: 0.9447259217165278at epoch 7\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Epoch:  40%|████      | 8/20 [2:34:28<3:51:44, 1158.67s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "        B_nr       0.98      0.97      0.97      4839\n",
      "        B_ns       0.96      0.94      0.95      5576\n",
      "        B_nt       0.92      0.95      0.94      2695\n",
      "        E_nr       0.96      0.96      0.96      4838\n",
      "        E_ns       0.95      0.91      0.93      5574\n",
      "        E_nt       0.93      0.94      0.93      2692\n",
      "        M_nr       0.97      0.98      0.97      4382\n",
      "        M_ns       0.84      0.89      0.87      3453\n",
      "        M_nt       0.91      0.96      0.93     10073\n",
      "           O       0.99      0.99      0.99     96582\n",
      "\n",
      "    accuracy                           0.97    140704\n",
      "   macro avg       0.94      0.95      0.94    140704\n",
      "weighted avg       0.97      0.97      0.97    140704\n",
      "\n",
      "\n",
      "Epoch: 8, Step: 0, Train loss: 0.02617947943508625\n",
      "Epoch: 8, Step: 100, Train loss: 0.03863345995420903\n",
      "Epoch: 8, Step: 200, Train loss: 0.035441361563175976\n",
      "Epoch: 8, Step: 300, Train loss: 0.035436487860515085\n",
      "Epoch: 8, Step: 400, Train loss: 0.034275056143985724\n",
      "Epoch: 8, Step: 500, Train loss: 0.03380340462905576\n",
      "Epoch: 8, Step: 600, Train loss: 0.034512639737427665\n",
      "Epoch: 8, Step: 700, Train loss: 0.03431218536886202\n",
      "Epoch: 8, Step: 800, Train loss: 0.034289067102707786\n",
      "Epoch: 8, Average train loss: 0.0346592340890226 \n",
      "Validation loss: 0.14098035336569303 at epoch 8\n",
      "Validation Accuracy: 0.9707897430065954at epoch 8\n",
      "Validation F1-Score: 0.9402778294319972at epoch 8\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Epoch:  45%|████▌     | 9/20 [2:53:46<3:32:20, 1158.25s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "        B_nr       0.98      0.96      0.97      4839\n",
      "        B_ns       0.94      0.95      0.95      5576\n",
      "        B_nt       0.94      0.91      0.92      2695\n",
      "        E_nr       0.97      0.96      0.96      4838\n",
      "        E_ns       0.92      0.93      0.92      5574\n",
      "        E_nt       0.94      0.89      0.92      2692\n",
      "        M_nr       0.97      0.97      0.97      4382\n",
      "        M_ns       0.93      0.83      0.87      3453\n",
      "        M_nt       0.94      0.90      0.92     10073\n",
      "           O       0.98      0.99      0.99     96582\n",
      "\n",
      "    accuracy                           0.97    140704\n",
      "   macro avg       0.95      0.93      0.94    140704\n",
      "weighted avg       0.97      0.97      0.97    140704\n",
      "\n",
      "\n",
      "Epoch: 9, Step: 0, Train loss: 0.0999036505818367\n",
      "Epoch: 9, Step: 100, Train loss: 0.02913193331021668\n",
      "Epoch: 9, Step: 200, Train loss: 0.029320975582564798\n",
      "Epoch: 9, Step: 300, Train loss: 0.030679794494256042\n",
      "Epoch: 9, Step: 400, Train loss: 0.03002985310203691\n",
      "Epoch: 9, Step: 500, Train loss: 0.029123012110555613\n",
      "Epoch: 9, Step: 600, Train loss: 0.028650679489204075\n",
      "Epoch: 9, Step: 700, Train loss: 0.028785426864140106\n",
      "Epoch: 9, Step: 800, Train loss: 0.02916083896665493\n",
      "Epoch: 9, Average train loss: 0.028670835605399795 \n",
      "Validation loss: 0.1342156738731132 at epoch 9\n",
      "Validation Accuracy: 0.9744214805549238at epoch 9\n",
      "Validation F1-Score: 0.9472142215089766at epoch 9\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Epoch:  50%|█████     | 10/20 [3:13:03<3:12:59, 1158.00s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "        B_nr       0.98      0.98      0.98      4839\n",
      "        B_ns       0.95      0.95      0.95      5576\n",
      "        B_nt       0.93      0.94      0.94      2695\n",
      "        E_nr       0.96      0.97      0.96      4838\n",
      "        E_ns       0.94      0.93      0.94      5574\n",
      "        E_nt       0.92      0.95      0.93      2692\n",
      "        M_nr       0.96      0.98      0.97      4382\n",
      "        M_ns       0.89      0.87      0.88      3453\n",
      "        M_nt       0.92      0.95      0.93     10073\n",
      "           O       0.99      0.99      0.99     96582\n",
      "\n",
      "    accuracy                           0.97    140704\n",
      "   macro avg       0.94      0.95      0.95    140704\n",
      "weighted avg       0.97      0.97      0.97    140704\n",
      "\n",
      "\n",
      "Epoch: 10, Step: 0, Train loss: 0.002057593548670411\n",
      "Epoch: 10, Step: 100, Train loss: 0.016354165911342543\n",
      "Epoch: 10, Step: 200, Train loss: 0.019615491087356145\n",
      "Epoch: 10, Step: 300, Train loss: 0.020212128447667323\n",
      "Epoch: 10, Step: 400, Train loss: 0.021578719068383635\n",
      "Epoch: 10, Step: 500, Train loss: 0.022916066180310488\n",
      "Epoch: 10, Step: 600, Train loss: 0.023766184421996128\n",
      "Epoch: 10, Step: 700, Train loss: 0.023925366843007664\n",
      "Epoch: 10, Step: 800, Train loss: 0.024078985378954247\n",
      "Epoch: 10, Average train loss: 0.023748882391291783 \n",
      "Validation loss: 0.13431897253285693 at epoch 10\n",
      "Validation Accuracy: 0.9729218785535593at epoch 10\n",
      "Validation F1-Score: 0.9446762478343572at epoch 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Epoch:  55%|█████▌    | 11/20 [3:32:21<2:53:42, 1158.06s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "        B_nr       0.98      0.97      0.98      4839\n",
      "        B_ns       0.93      0.96      0.94      5576\n",
      "        B_nt       0.95      0.93      0.94      2695\n",
      "        E_nr       0.97      0.96      0.96      4838\n",
      "        E_ns       0.89      0.95      0.92      5574\n",
      "        E_nt       0.94      0.93      0.94      2692\n",
      "        M_nr       0.97      0.97      0.97      4382\n",
      "        M_ns       0.85      0.91      0.88      3453\n",
      "        M_nt       0.94      0.93      0.93     10073\n",
      "           O       0.99      0.99      0.99     96582\n",
      "\n",
      "    accuracy                           0.97    140704\n",
      "   macro avg       0.94      0.95      0.94    140704\n",
      "weighted avg       0.97      0.97      0.97    140704\n",
      "\n",
      "\n",
      "Epoch: 11, Step: 0, Train loss: 0.04864921420812607\n",
      "Epoch: 11, Step: 100, Train loss: 0.020617362255308923\n",
      "Epoch: 11, Step: 200, Train loss: 0.01870826850106606\n",
      "Epoch: 11, Step: 300, Train loss: 0.018346095874602366\n",
      "Epoch: 11, Step: 400, Train loss: 0.018905554982128685\n",
      "Epoch: 11, Step: 500, Train loss: 0.019377154734045148\n",
      "Epoch: 11, Step: 600, Train loss: 0.019967921781442757\n",
      "Epoch: 11, Step: 700, Train loss: 0.019901650350734514\n",
      "Epoch: 11, Step: 800, Train loss: 0.019781829602522227\n",
      "Epoch: 11, Average train loss: 0.01966104546138342 \n",
      "Validation loss: 0.13417657301251648 at epoch 11\n",
      "Validation Accuracy: 0.9744570161473732at epoch 11\n",
      "Validation F1-Score: 0.9472400661291183at epoch 11\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Epoch:  60%|██████    | 12/20 [3:51:39<2:34:23, 1157.98s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "        B_nr       0.98      0.97      0.98      4839\n",
      "        B_ns       0.96      0.94      0.95      5576\n",
      "        B_nt       0.93      0.95      0.94      2695\n",
      "        E_nr       0.98      0.96      0.97      4838\n",
      "        E_ns       0.94      0.92      0.93      5574\n",
      "        E_nt       0.94      0.93      0.93      2692\n",
      "        M_nr       0.97      0.98      0.97      4382\n",
      "        M_ns       0.84      0.92      0.88      3453\n",
      "        M_nt       0.92      0.95      0.93     10073\n",
      "           O       0.99      0.99      0.99     96582\n",
      "\n",
      "    accuracy                           0.97    140704\n",
      "   macro avg       0.94      0.95      0.95    140704\n",
      "weighted avg       0.97      0.97      0.97    140704\n",
      "\n",
      "\n",
      "Epoch: 12, Step: 0, Train loss: 0.03301943838596344\n",
      "Epoch: 12, Step: 100, Train loss: 0.015860037982756602\n",
      "Epoch: 12, Step: 200, Train loss: 0.017225448519109624\n",
      "Epoch: 12, Step: 300, Train loss: 0.01650072940825593\n",
      "Epoch: 12, Step: 400, Train loss: 0.015527181533900347\n",
      "Epoch: 12, Step: 500, Train loss: 0.01606742256736251\n",
      "Epoch: 12, Step: 600, Train loss: 0.016172683219670997\n",
      "Epoch: 12, Step: 700, Train loss: 0.01714767568137566\n",
      "Epoch: 12, Step: 800, Train loss: 0.017138846590359243\n",
      "Epoch: 12, Average train loss: 0.017268539859940264 \n",
      "Validation loss: 0.14243990918419555 at epoch 12\n",
      "Validation Accuracy: 0.975409370025017at epoch 12\n",
      "Validation F1-Score: 0.9493121523720532at epoch 12\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Epoch:  65%|██████▌   | 13/20 [4:10:57<2:15:05, 1157.87s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "        B_nr       0.98      0.98      0.98      4839\n",
      "        B_ns       0.96      0.94      0.95      5576\n",
      "        B_nt       0.93      0.95      0.94      2695\n",
      "        E_nr       0.97      0.97      0.97      4838\n",
      "        E_ns       0.95      0.92      0.93      5574\n",
      "        E_nt       0.93      0.95      0.94      2692\n",
      "        M_nr       0.97      0.98      0.98      4382\n",
      "        M_ns       0.92      0.85      0.88      3453\n",
      "        M_nt       0.93      0.94      0.94     10073\n",
      "           O       0.99      0.99      0.99     96582\n",
      "\n",
      "    accuracy                           0.98    140704\n",
      "   macro avg       0.95      0.95      0.95    140704\n",
      "weighted avg       0.98      0.98      0.98    140704\n",
      "\n",
      "\n",
      "Epoch: 13, Step: 0, Train loss: 0.000479490146972239\n",
      "Epoch: 13, Step: 100, Train loss: 0.015417802542342002\n",
      "Epoch: 13, Step: 200, Train loss: 0.014208690174281903\n",
      "Epoch: 13, Step: 300, Train loss: 0.01347711198876977\n",
      "Epoch: 13, Step: 400, Train loss: 0.013607595478718116\n",
      "Epoch: 13, Step: 500, Train loss: 0.013398655516384987\n",
      "Epoch: 13, Step: 600, Train loss: 0.012684930426443612\n",
      "Epoch: 13, Step: 700, Train loss: 0.013013297505112107\n",
      "Epoch: 13, Step: 800, Train loss: 0.012925670167453617\n",
      "Epoch: 13, Average train loss: 0.013149122373277297 \n",
      "Validation loss: 0.13477735228676613 at epoch 13\n",
      "Validation Accuracy: 0.9751677279963612at epoch 13\n",
      "Validation F1-Score: 0.948728619766259at epoch 13\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Epoch:  70%|███████   | 14/20 [4:30:15<1:55:48, 1158.04s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "        B_nr       0.98      0.98      0.98      4839\n",
      "        B_ns       0.94      0.96      0.95      5576\n",
      "        B_nt       0.95      0.93      0.94      2695\n",
      "        E_nr       0.96      0.97      0.97      4838\n",
      "        E_ns       0.92      0.95      0.93      5574\n",
      "        E_nt       0.93      0.94      0.94      2692\n",
      "        M_nr       0.97      0.98      0.98      4382\n",
      "        M_ns       0.88      0.89      0.89      3453\n",
      "        M_nt       0.94      0.92      0.93     10073\n",
      "           O       0.99      0.99      0.99     96582\n",
      "\n",
      "    accuracy                           0.98    140704\n",
      "   macro avg       0.95      0.95      0.95    140704\n",
      "weighted avg       0.98      0.98      0.98    140704\n",
      "\n",
      "\n",
      "Epoch: 14, Step: 0, Train loss: 0.0015078940195962787\n",
      "Epoch: 14, Step: 100, Train loss: 0.012611548313588633\n",
      "Epoch: 14, Step: 200, Train loss: 0.011818829071748567\n",
      "Epoch: 14, Step: 300, Train loss: 0.010764713678741053\n",
      "Epoch: 14, Step: 400, Train loss: 0.010983451294606468\n",
      "Epoch: 14, Step: 500, Train loss: 0.011009359975179756\n",
      "Epoch: 14, Step: 600, Train loss: 0.011443479174149328\n",
      "Epoch: 14, Step: 700, Train loss: 0.010998044687023681\n",
      "Epoch: 14, Step: 800, Train loss: 0.011495195712535072\n",
      "Epoch: 14, Average train loss: 0.011653032864826203 \n",
      "Validation loss: 0.13007254127757184 at epoch 14\n",
      "Validation Accuracy: 0.9763048669547418at epoch 14\n",
      "Validation F1-Score: 0.952075073320174at epoch 14\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Epoch:  75%|███████▌  | 15/20 [4:49:33<1:36:29, 1157.92s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "        B_nr       0.98      0.98      0.98      4839\n",
      "        B_ns       0.96      0.95      0.95      5576\n",
      "        B_nt       0.95      0.95      0.95      2695\n",
      "        E_nr       0.98      0.96      0.97      4838\n",
      "        E_ns       0.94      0.94      0.94      5574\n",
      "        E_nt       0.94      0.94      0.94      2692\n",
      "        M_nr       0.97      0.98      0.98      4382\n",
      "        M_ns       0.90      0.89      0.89      3453\n",
      "        M_nt       0.94      0.93      0.94     10073\n",
      "           O       0.99      0.99      0.99     96582\n",
      "\n",
      "    accuracy                           0.98    140704\n",
      "   macro avg       0.95      0.95      0.95    140704\n",
      "weighted avg       0.98      0.98      0.98    140704\n",
      "\n",
      "\n",
      "Epoch: 15, Step: 0, Train loss: 0.0005320027121342719\n",
      "Epoch: 15, Step: 100, Train loss: 0.009322206869820538\n",
      "Epoch: 15, Step: 200, Train loss: 0.009169618311987968\n",
      "Epoch: 15, Step: 300, Train loss: 0.010665729227499027\n",
      "Epoch: 15, Step: 400, Train loss: 0.011127407766357204\n",
      "Epoch: 15, Step: 500, Train loss: 0.010979953316520637\n",
      "Epoch: 15, Step: 600, Train loss: 0.010846796468405452\n",
      "Epoch: 15, Step: 700, Train loss: 0.010745811636053815\n",
      "Epoch: 15, Step: 800, Train loss: 0.01040040629378653\n",
      "Epoch: 15, Average train loss: 0.00998326052257611 \n",
      "Validation loss: 0.13508997763331074 at epoch 15\n",
      "Validation Accuracy: 0.9768805435524222at epoch 15\n",
      "Validation F1-Score: 0.9524518447831655at epoch 15\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Epoch:  80%|████████  | 16/20 [5:08:50<1:17:10, 1157.72s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "        B_nr       0.98      0.98      0.98      4839\n",
      "        B_ns       0.95      0.95      0.95      5576\n",
      "        B_nt       0.93      0.95      0.94      2695\n",
      "        E_nr       0.97      0.97      0.97      4838\n",
      "        E_ns       0.93      0.94      0.94      5574\n",
      "        E_nt       0.94      0.94      0.94      2692\n",
      "        M_nr       0.97      0.98      0.98      4382\n",
      "        M_ns       0.91      0.88      0.89      3453\n",
      "        M_nt       0.94      0.95      0.94     10073\n",
      "           O       0.99      0.99      0.99     96582\n",
      "\n",
      "    accuracy                           0.98    140704\n",
      "   macro avg       0.95      0.95      0.95    140704\n",
      "weighted avg       0.98      0.98      0.98    140704\n",
      "\n",
      "\n",
      "Epoch: 16, Step: 0, Train loss: 0.0003239345969632268\n",
      "Epoch: 16, Step: 100, Train loss: 0.007251097451210239\n",
      "Epoch: 16, Step: 200, Train loss: 0.009396115280692154\n",
      "Epoch: 16, Step: 300, Train loss: 0.009491230373274097\n",
      "Epoch: 16, Step: 400, Train loss: 0.00935119491808556\n",
      "Epoch: 16, Step: 500, Train loss: 0.008507968484661184\n",
      "Epoch: 16, Step: 600, Train loss: 0.008221835992330498\n",
      "Epoch: 16, Step: 700, Train loss: 0.008335276821160476\n",
      "Epoch: 16, Step: 800, Train loss: 0.008620695990828792\n",
      "Epoch: 16, Average train loss: 0.008232681609388543 \n",
      "Validation loss: 0.14699454703251144 at epoch 16\n",
      "Validation Accuracy: 0.9763830452581306at epoch 16\n",
      "Validation F1-Score: 0.9517859745423545at epoch 16\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Epoch:  85%|████████▌ | 17/20 [5:28:08<57:53, 1157.71s/it]  "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "        B_nr       0.98      0.98      0.98      4839\n",
      "        B_ns       0.95      0.95      0.95      5576\n",
      "        B_nt       0.93      0.95      0.94      2695\n",
      "        E_nr       0.97      0.96      0.97      4838\n",
      "        E_ns       0.94      0.94      0.94      5574\n",
      "        E_nt       0.94      0.94      0.94      2692\n",
      "        M_nr       0.97      0.98      0.98      4382\n",
      "        M_ns       0.90      0.89      0.89      3453\n",
      "        M_nt       0.93      0.95      0.94     10073\n",
      "           O       0.99      0.99      0.99     96582\n",
      "\n",
      "    accuracy                           0.98    140704\n",
      "   macro avg       0.95      0.95      0.95    140704\n",
      "weighted avg       0.98      0.98      0.98    140704\n",
      "\n",
      "\n",
      "Epoch: 17, Step: 0, Train loss: 0.020014125853776932\n",
      "Epoch: 17, Step: 100, Train loss: 0.005708400528842818\n",
      "Epoch: 17, Step: 200, Train loss: 0.006152419265122466\n",
      "Epoch: 17, Step: 300, Train loss: 0.0058768058959419835\n",
      "Epoch: 17, Step: 400, Train loss: 0.005958253484143251\n",
      "Epoch: 17, Step: 500, Train loss: 0.006401044306438347\n",
      "Epoch: 17, Step: 600, Train loss: 0.006487303643351608\n",
      "Epoch: 17, Step: 700, Train loss: 0.006807552822733935\n",
      "Epoch: 17, Step: 800, Train loss: 0.006878792363226666\n",
      "Epoch: 17, Average train loss: 0.007042284748767159 \n",
      "Validation loss: 0.14682144997803132 at epoch 17\n",
      "Validation Accuracy: 0.9768805435524222at epoch 17\n",
      "Validation F1-Score: 0.9523913324392017at epoch 17\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Epoch:  90%|█████████ | 18/20 [5:47:26<38:35, 1157.74s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "        B_nr       0.98      0.98      0.98      4839\n",
      "        B_ns       0.96      0.95      0.95      5576\n",
      "        B_nt       0.93      0.96      0.94      2695\n",
      "        E_nr       0.97      0.97      0.97      4838\n",
      "        E_ns       0.94      0.93      0.94      5574\n",
      "        E_nt       0.93      0.95      0.94      2692\n",
      "        M_nr       0.97      0.98      0.98      4382\n",
      "        M_ns       0.91      0.88      0.89      3453\n",
      "        M_nt       0.93      0.95      0.94     10073\n",
      "           O       0.99      0.99      0.99     96582\n",
      "\n",
      "    accuracy                           0.98    140704\n",
      "   macro avg       0.95      0.95      0.95    140704\n",
      "weighted avg       0.98      0.98      0.98    140704\n",
      "\n",
      "\n",
      "Epoch: 18, Step: 0, Train loss: 0.0001083395181922242\n",
      "Epoch: 18, Step: 100, Train loss: 0.004615276137532293\n",
      "Epoch: 18, Step: 200, Train loss: 0.004288479154095498\n",
      "Epoch: 18, Step: 300, Train loss: 0.005680499854976229\n",
      "Epoch: 18, Step: 400, Train loss: 0.005951523773746386\n",
      "Epoch: 18, Step: 500, Train loss: 0.005820361774834431\n",
      "Epoch: 18, Step: 600, Train loss: 0.006224244637062274\n",
      "Epoch: 18, Step: 700, Train loss: 0.00614507974635447\n",
      "Epoch: 18, Step: 800, Train loss: 0.006310585403546982\n",
      "Epoch: 18, Average train loss: 0.006325741246509228 \n",
      "Validation loss: 0.14264975422790457 at epoch 18\n",
      "Validation Accuracy: 0.9772714350693654at epoch 18\n",
      "Validation F1-Score: 0.953518068041436at epoch 18\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Epoch:  95%|█████████▌| 19/20 [6:06:43<19:17, 1157.74s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "        B_nr       0.98      0.98      0.98      4839\n",
      "        B_ns       0.96      0.95      0.95      5576\n",
      "        B_nt       0.94      0.96      0.95      2695\n",
      "        E_nr       0.97      0.97      0.97      4838\n",
      "        E_ns       0.94      0.94      0.94      5574\n",
      "        E_nt       0.94      0.95      0.94      2692\n",
      "        M_nr       0.97      0.98      0.98      4382\n",
      "        M_ns       0.92      0.88      0.90      3453\n",
      "        M_nt       0.94      0.95      0.94     10073\n",
      "           O       0.99      0.99      0.99     96582\n",
      "\n",
      "    accuracy                           0.98    140704\n",
      "   macro avg       0.95      0.95      0.95    140704\n",
      "weighted avg       0.98      0.98      0.98    140704\n",
      "\n",
      "\n",
      "Epoch: 19, Step: 0, Train loss: 0.00011570870992727578\n",
      "Epoch: 19, Step: 100, Train loss: 0.006113917242947708\n",
      "Epoch: 19, Step: 200, Train loss: 0.006143870602468427\n",
      "Epoch: 19, Step: 300, Train loss: 0.006183979809760894\n",
      "Epoch: 19, Step: 400, Train loss: 0.006107402420491694\n",
      "Epoch: 19, Step: 500, Train loss: 0.005579850703983023\n",
      "Epoch: 19, Step: 600, Train loss: 0.0055224724435875075\n",
      "Epoch: 19, Step: 700, Train loss: 0.005377371983716235\n",
      "Epoch: 19, Step: 800, Train loss: 0.005174442818508609\n",
      "Epoch: 19, Average train loss: 0.005458298573347603 \n",
      "Validation loss: 0.14362447592559657 at epoch 19\n",
      "Validation Accuracy: 0.9773851489652036at epoch 19\n",
      "Validation F1-Score: 0.953945534370719at epoch 19\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epoch: 100%|██████████| 20/20 [6:26:01<00:00, 1158.06s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "        B_nr       0.98      0.98      0.98      4839\n",
      "        B_ns       0.96      0.95      0.96      5576\n",
      "        B_nt       0.94      0.95      0.95      2695\n",
      "        E_nr       0.97      0.97      0.97      4838\n",
      "        E_ns       0.94      0.94      0.94      5574\n",
      "        E_nt       0.94      0.95      0.94      2692\n",
      "        M_nr       0.97      0.98      0.98      4382\n",
      "        M_ns       0.91      0.88      0.90      3453\n",
      "        M_nt       0.94      0.95      0.94     10073\n",
      "           O       0.99      0.99      0.99     96582\n",
      "\n",
      "    accuracy                           0.98    140704\n",
      "   macro avg       0.95      0.95      0.95    140704\n",
      "weighted avg       0.98      0.98      0.98    140704\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from tqdm import tqdm, trange\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "## 记录每一周次训练完的平均损失和验证损失\n",
    "loss_values, validation_loss_values = [], []\n",
    "\n",
    "for epoch in trange(epochs, desc=\"Epoch\"):\n",
    "    # ========================================\n",
    "    #               训练\n",
    "    # ========================================\n",
    "\n",
    "    #  训练模式\n",
    "    model.train()\n",
    "\n",
    "    # 损失\n",
    "    total_loss = 0\n",
    "\n",
    "    # 训练循环\n",
    "    for step, batch in enumerate(train_dataloader):\n",
    "        # 数据 gpu\n",
    "        batch = tuple(t.to(device) for t in batch)\n",
    "        b_input_ids, b_input_mask, b_labels = batch\n",
    "\n",
    "        # 梯度清零\n",
    "        model.zero_grad()\n",
    "\n",
    "        # 前向计算，获得损失\n",
    "        outputs = model(b_input_ids,\n",
    "                        token_type_ids=None,\n",
    "                        attention_mask=b_input_mask,\n",
    "                        labels=b_labels)\n",
    "        loss = outputs[0]\n",
    "\n",
    "        # 反向传播\n",
    "        loss.backward()\n",
    "\n",
    "        # 累加损失\n",
    "        total_loss += loss.item()\n",
    "\n",
    "        # 梯度裁剪，防止梯度爆炸\n",
    "        torch.nn.utils.clip_grad_norm_(parameters=model.parameters(),\n",
    "                                       max_norm=max_grad_norm)\n",
    "        # 更新参数\n",
    "        optimizer.step()\n",
    "        # 更新学习率\n",
    "        scheduler.step()\n",
    "\n",
    "        if step % 100 == 0:\n",
    "            print(\"Epoch: {}, Step: {}, Train loss: {}\".format(\n",
    "                epoch, step, total_loss / (step + 1)))\n",
    "        \n",
    "\n",
    "    # 计算每一训练循环的平均损失\n",
    "    avg_train_loss = total_loss / len(train_dataloader)\n",
    "    print(\"Epoch: {}, Average train loss: {} \".format(epoch, avg_train_loss))\n",
    "\n",
    "    loss_values.append(avg_train_loss)\n",
    "\n",
    "    # ========================================\n",
    "    #               验证\n",
    "    # ========================================\n",
    "\n",
    "    # 验证模式\n",
    "    model.eval()\n",
    "\n",
    "    # 验证损失及验证精度\n",
    "    eval_loss, eval_accuracy = 0, 0\n",
    "\n",
    "    nb_eval_steps, nb_eval_examples = 0, 0\n",
    "\n",
    "    predictions, true_labels = [], []\n",
    "    for batch in valid_dataloader:\n",
    "        batch = tuple(t.to(device) for t in batch)\n",
    "        b_input_ids, b_input_mask, b_labels = batch\n",
    "\n",
    "        # 验证时，不更新梯度\n",
    "        with torch.no_grad():\n",
    "\n",
    "            outputs = model(\n",
    "                b_input_ids,\n",
    "                token_type_ids=None,\n",
    "                attention_mask=b_input_mask,\n",
    "                labels=b_labels,  # 没有提供标签值，返回值为 权重分布，而不是损失值\n",
    "            )\n",
    "\n",
    "        # 数据移动到 cpu 上\n",
    "        logits = outputs[1].detach().cpu()\n",
    "        label_ids = b_labels.to('cpu')\n",
    "        b_input_mask = b_input_mask.to('cpu')\n",
    "\n",
    "        # 累加损失值\n",
    "        eval_loss += outputs[0].mean().item()\n",
    "\n",
    "        # 预测标签\n",
    "        b_preds = torch.argmax(logits, dim=2)\n",
    "\n",
    "        predictions.append(b_preds.masked_select(b_input_mask.bool()))\n",
    "        true_labels.append(label_ids.masked_select(b_input_mask.bool()))\n",
    "\n",
    "    eval_loss = eval_loss / len(valid_dataloader)\n",
    "    validation_loss_values.append(eval_loss)\n",
    "    print(\"Validation loss: {} at epoch {}\".format(eval_loss, epoch))\n",
    "    \n",
    "    predictions = torch.cat(predictions)\n",
    "    true_labels = torch.cat(true_labels)\n",
    "    \n",
    "\n",
    "    \n",
    "# 计算精度\n",
    "    pred_tags = [id2tag[idx] for idx in predictions.tolist()]\n",
    "    valid_tags = [id2tag[idx] for idx in true_labels.tolist()]\n",
    "    print(\"Validation Accuracy: {}at epoch {}\".format(\n",
    "        accuracy_score(valid_tags, pred_tags), epoch))\n",
    "    print(\"Validation F1-Score: {}at epoch {}\".format(\n",
    "        f1_score(valid_tags, pred_tags, average='macro'), epoch))\n",
    "    valid_report = classification_report(valid_tags, pred_tags)\n",
    "    print(valid_report)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 从整个训练过程来看，比较难训练，但最终准确率较高"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T13:23:50.253407Z",
     "start_time": "2020-05-29T13:23:50.011196Z"
    }
   },
   "outputs": [],
   "source": [
    "# # 保存预训练模型\n",
    "# model.save_pretrained(\"../../H/models/huggingface/torch/zh_albert_ner/\")\n",
    "torch.save(model.state_dict(),\n",
    "           \"../../H/models/huggingface/torch/zh_albert_ner/pytorch_model.bin\")\n",
    "\n",
    "# # 直接从路径中加载模型\n",
    "# model_path = \"../../H/models/huggingface/torch/zh_albert_ner/\"\n",
    "# model = AlbertForTokenClassification.from_pretrained(model_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T13:14:25.642154Z",
     "start_time": "2020-05-29T13:14:25.532710Z"
    }
   },
   "outputs": [],
   "source": [
    "input_ids = torch.tensor(\n",
    "    tokenizer.encode(\"冯永祥突发奇想，跑到阿尔及利亚旅行，意外结识了印度人民党的领导\",\n",
    "                     add_special_tokens=False)).unsqueeze(0).to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T13:14:26.694748Z",
     "start_time": "2020-05-29T13:14:26.692073Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1101, 3719, 4872, 4960, 1355, 1936, 2682, 8024, 6651, 1168, 7350, 2209,\n",
       "         1350, 1164,  762, 3180, 6121, 8024, 2692, 1912, 5310, 6399,  749, 1313,\n",
       "         2428,  782, 3696, 1054, 4638, 7566, 2193]], device='cuda:0')"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T13:14:28.739423Z",
     "start_time": "2020-05-29T13:14:28.711554Z"
    }
   },
   "outputs": [],
   "source": [
    "outputs = model(input_ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T13:14:29.870385Z",
     "start_time": "2020-05-29T13:14:29.866923Z"
    }
   },
   "outputs": [],
   "source": [
    "pred_dis = outputs[0]\n",
    "pred = torch.argmax(pred_dis, axis=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T13:14:31.096912Z",
     "start_time": "2020-05-29T13:14:31.093298Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['B_nr',\n",
       " 'M_nr',\n",
       " 'E_nr',\n",
       " 'O',\n",
       " 'O',\n",
       " 'O',\n",
       " 'O',\n",
       " 'O',\n",
       " 'O',\n",
       " 'O',\n",
       " 'B_ns',\n",
       " 'M_ns',\n",
       " 'M_ns',\n",
       " 'M_ns',\n",
       " 'E_ns',\n",
       " 'O',\n",
       " 'O',\n",
       " 'O',\n",
       " 'O',\n",
       " 'O',\n",
       " 'O',\n",
       " 'O',\n",
       " 'O',\n",
       " 'B_nt',\n",
       " 'M_nt',\n",
       " 'M_nt',\n",
       " 'M_nt',\n",
       " 'E_nt',\n",
       " 'O',\n",
       " 'O',\n",
       " 'O']"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[id2tag[idx] for idx in pred[0].cpu().numpy()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-29T13:31:19.774111Z",
     "start_time": "2020-05-29T13:31:19.664246Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[3, 4, 5, 9, 9, 9, 9, 9, 9, 9, 0, 1, 1, 1, 2, 9, 9, 9, 9, 9, 9, 9, 9, 6,\n",
       "         7, 7, 7, 8, 9, 9, 9]], device='cuda:0')"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "position": {
    "height": "1040.5px",
    "left": "710px",
    "right": "20px",
    "top": "120px",
    "width": "350px"
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
